The field of the invention is image enhancement and more particularly image enhancement based on object profiling and histogram signature.
FIG. 1 illustrates a prior art binocular type digital imaging system and displays the problem of interest. The object to be identified 100 may appear anywhere in the picture. This object may have various sizes (in the image) depending on its distance from the camera 101 or other information gathering device which provides the source of the visual rendering. Also, the object may be rotated, so its orientation may change. Both the lightness level and the contrast of the object to be discerned may also be compromised 102 yet the operator has to make a quick decision on the identity of the entity in question.
FIG. 2 illustrates an example of a prior art head mounted digital imaging system. In FIG. 2, the operator is wearing a helmet-mounted display or eye-glass system 200. This display may be a night vision goggle device to amplify light or other apparatus to enable the user to discern better quality information from the environment yet still be able to move around. In FIG. 2, the aircraft 202, or other object to be examined, impacts the visual display at the multi-spectral sensors at 201. These data are then transferred to the central processing unit (CPU) 202, which may consist of a chip or other microprocessor element. Also, at the central processor, is a signal received from the output of the adjustment knob 204. After the identification of the object is made more salient, the final output signal to the HMD (head mounted display) is transferred from the CPU to the HMD 205. The signal is further transferred to the optical glass at 206 as indicated in FIG. 2. The operator sees the modified image at the optical glass. There may exist a latency in producing the enhanced image at 206 because of the computation time and cycle time to process the requisite information (going from points 201-206).
Prior Art image recognition methods may employ a difference matrix D. In order to find an object Ak when it is embedded in a large picture, you must take a sample matrix Am. In other words, the goal will be to find a submatrix Ak (the same size of Am) when it may appear anywhere in the larger matrix A, which is searched. A difference matrix D=Am−Ak is computed between the sample and the object. If the difference matrix is essentially filled with zero elements, then the test image Ak has been “found” in the larger image at the spatial location of the coordinates of Am and the object has been identified.
Some problems that often occur with prior art image recognition include:
The computational burden of computing all the pixels that must be used in Am and Ak;
Noise may enter the image and bias the results. The present invention will address noise in images;
The size of the object may be different from that initially planned. The training image Ak may be of the incorrect size due to inaccurate assumptions of camera distance;
The sensitivity of detecting when ∥D∥2 is near zero is inadequate in prior art systems; and
The sample object Am in the data image may be rotated and appear significantly different from the initially trained image Ak. Consequently, there will not be a perfect match due to an orientation change of the sampled object. In United States Air Force applications, many pictures are taken from the air and the object to be examined may have an arbitrary rotation, which cannot be known apriori. The present invention employs an image identification algorithm sufficiently robust that does not depend on the orientation of the object nor on the relative size of the object.
Such prior art deficiencies with respect to rotation are illustrated in FIG. 5. FIG. 5 portrays an aircraft object (Ak) 500 we wish to, identify in a larger image. For simplicity, it will be assumed that the sizes (number of rows, columns and pixels of the sample matrix Am and object matrix Ak) are known and fixed. FIG. 6 shows the first data test image (A1) constructed to examine the efficacy of the present methods to identify the object Ak. Please note in FIG. 6, the three sites of the test objects, 602-604, have the identical orientation as the trained object Ak 500 in FIG. 5. FIGS. 7a and 7b show the plots of the L2 norm (σ1) for the top 600 and bottom 601 rows of FIG. 6. In FIGS. 7a and 7b, the x-axis, 701 and 703, respectively, is the spatial location in the image, and the y-axis, 700 and 702 is L2 norm (σ1). Since sudden downward changes of σ1 are an indication of the object in the picture, clearly this norm has efficacy of identifying the spatial location of objects because of the sudden and sharp dips of the L2 norm at the spatial location where the object appeared. In FIG. 7a the dip representing the location where the object appeared is at 705 for the top row of FIG. 6 and in the 7b plot 704 represents the location where the airplanes appear in the bottom row of FIG. 6.
In FIGS. 8a and 8b, the same results are shown with the Frobenius norm. The Frobenius norm refers to the sums of the squares of all the elements of a matrix. For a difference matrix (which has most terms near zero), the sums of the squares of all the elements is a small positive number. In FIGS. 8a and 8b the x-axis at 801 and 804 respectively is the spatial location in the image. The image has 700 columns from left to right and the x-axis is the bin or column number and the y-axis at 800 and 803 represents Frobenius norm. Each bin number is 1/700 the distance from the left to the right side of the image. Again the spatial location of the object can be identified because the Frobenius norm suddenly drops to a low value at the true location of the object, at 802 in FIG. 8a and at 805 in FIG. 8b. It is significant that the results in FIGS. 7 and 8 look impressive because the trained object was not rotated from the original test image supplied in FIG. 5.
To further illustrate the deficiencies in the prior art with rotation, the test object is now rotated and inserted into a second test image A2 as shown in FIG. 9. The goal is to correctly detect the spatial position in the image A2 where the object may appear. FIG. 10 displays the plot of the L2 norm and FIG. 11 shows the corresponding plot of the Frobenius norm for the test image A2. This prior art method has failed to correctly identify the spatial location of the object. FIGS. 10a, 10b and 11a and 11b don't have a distinct sudden drop or dip to indicate location of an object as do FIGS. 7a, 7b, 8a and 8b. It is clear now that because the trained image did not contain rotational information about the object in its initialization, prior art methods are ineffective in identifying objects in a test image. They break down when the object appears in the data image but has been rotated. This present invention provides means for identifying an object in a visual image having size and rotation variations and other viewing issues in the prior art.